Machine-learning-based techniques for determining response team predictions for incident alerts in a complex platform

ABSTRACT

Various embodiments of the present invention provide methods, apparatuses, systems, computing devices, and/or the like that are configured accurately and programmatically train a responder prediction machine learning model for generating response team predictions based on the systematic collection of one or more responder prediction training corpuses comprising one or more alert related datasets in a responder prediction server system. For example, the responder prediction server system may extract one or more alert attributes for each of the one or more alert related datasets for training one or more responder prediction machine learning models and/or one or more prioritization machine learning models. The responder prediction machine learning model and prioritization machine learning models may process one or more alerts, in real-time, to generate one or more response team prediction objects for rendering in a response team suggestion interface.

BACKGROUND

The complexity of enterprise software has matured to a degree that thereare now more potential failure points than ever. The impact of anincident can be devastating. Some estimates suggest that major incidentscan cost an organization $300,000 per hour that an enterprise softwaresystem is down. Applicant has identified many deficiencies and problemsassociated with existing methods, apparatuses, and systems forgenerating and transmitting response prediction alerts to initiateresponder action to address possible incidents in complex platform.Through applied effort, ingenuity, and innovation, these identifieddeficiencies and problems have been solved by developing solutions thatare embodied in accordance with the embodiments of the presentinvention, many examples of which are described in detail herein.

BRIEF SUMMARY

In general, embodiments of the present invention provide methods,apparatuses, systems, computing devices, and/or the like that areconfigured to effectively and efficiently extract one or more alertsfrom an alert monitoring service tool assigned to a complex platform;apply a responder prediction machine learning model to determine aresponse team prediction for each alert; determine, based on eachresponse team prediction, a client identifier set for of the responseteam prediction; and transmit the response team prediction to aprediction service API.

In accordance with another aspect, a computer-implemented method oftraining a responder prediction machine learning model for generatingresponse team predictions comprising: collecting alert related datasetsoriginating from one or more alert monitoring service tools over apredetermined time period; extracting alert attributes from the alertrelated datasets to create a responder prediction training corpus,wherein the alert attributes comprise an alert identifier, a tagidentifier, a log identifier, a description identifier, and a responderteam identifier; training the responder prediction machine learningmodel using the responder prediction training corpus; and storing theresponder prediction machine learning model following training to aresponder prediction model repository, wherein the responder predictionmodel repository is accessible by a responder prediction service.

In some embodiments, the computer-implemented method may furthercomprise: collecting second alert related datasets originating from theone or more alert monitoring service tools over a second predeterminedtime period; extracting second alert attributes from the second alertrelated datasets to create a second responder prediction trainingcorpus; training the responder prediction machine learning model usingthe second responder prediction training corpus; and storing theresponder prediction machine learning model following training to theresponder prediction model repository.

In some embodiments, the computer-implemented method may comprise:receiving one or more alerts from an alert monitoring service tool; andapplying, for each of the one or more alerts, a responder predictionmachine learning model to determine a response team prediction objectfor each alert. In some embodiments, the computer-implemented method maycomprise applying a score to each response team prediction object of theone or more alerts. In some embodiments, the computer-implemented methodmay comprise: determining the score of the response team predictionobject using at least one of a user input or a closing alert, andwherein the score is calculated by comparing the response teamprediction object with at least one of the user input or the closingalert. In some embodiments, the computer-implemented method may comprisetraining the responder prediction machine learning model in a subsequentstage using the score associated with each response team predictionobject of the one or more alerts. In some embodiments, thecomputer-implemented method is provided, wherein the score is applied tothe responder prediction machine learning model to determine one or morefuture response team prediction objects.

In some embodiments, the computer-implemented method may comprisetraining a prioritization machine learning model comprising: trainingthe prioritization machine learning model using the responder predictiontraining corpus, the alert attributes of the responder predictiontraining corpus further comprising a prioritization weight identifier;and storing the prioritization machine learning model following trainingto the responder prediction model repository, wherein the responderprediction model repository is accessible by a responder predictionservice. In some embodiments, the computer-implemented method maycomprise collecting second alert related datasets originating from theone or more alert monitoring service tools over a second predeterminedtime period; extracting second alert attributes from the second alertrelated datasets to create a second responder prediction trainingcorpus; training the prioritization machine learning model using thesecond responder prediction training corpus; and storing theprioritization machine learning model following training to theresponder prediction model repository.

In accordance with another aspect, an apparatus for generating aresponse team prediction associated with one or more alerts, theapparatus comprising at least one processor and at least one memoryincluding program code, the at least one memory and program codeconfigured to, with the processor, cause the apparatus to at least:receive one or more alerts from an alert monitoring service tool; foreach of the one or more alerts, apply a responder prediction machinelearning model to determine a response team prediction object for eachalert; and cause rendering of a response team suggestion interface basedon the response team prediction object.

In some embodiments, the apparatus is provided, wherein the responseteam prediction object is transmitted to a prediction service API thatis configured to indicate an alert notification comprising at least oneof the response team prediction, a dataset of routing informationassociated with at least a client identifier set for the response teamprediction, or the alert associated with the response team prediction.

In some embodiments, the apparatus is provided, wherein the responderprediction machine learning model comprises a pre-training with anextracted alert related dataset associated with a complex platform. Insome embodiments, the apparatus is provided, wherein the extracted alertrelated dataset comprises data extracted from a predetermined timeperiod.

In some embodiments, the apparatus is provided, wherein the at least onememory and program code configured to, with the processor, cause theapparatus to at least: receive one or more alerts from an alertmonitoring service tool; and for each of the one or more alerts, apply aprioritization machine learning model to determine a prioritizationweight for each alert. In some embodiments, the apparatus is provided,wherein an operation sequence of processing for the responder predictionmachine learning model is applied to the one or more alerts based on theprioritization weight for each of the one or more alerts. In someembodiments, the apparatus is provided, wherein an operation sequencefor determining the response team prediction object is applied to thealerts based on the prioritization weight for each alert. In someembodiments, the apparatus is provided, wherein an operation sequencefor the rendering of the response team suggestion interface based on theresponse team prediction object is based on the prioritization weightfor each of the one or more alerts used to generate the response teamprediction object.

In some embodiments, the apparatus is provided, wherein a score isdetermined by the response team prediction associated with an alert andat least one of user input or a closing alert. In some embodiments, theapparatus is provided, wherein the score is applied to the responderprediction machine learning model to determine one or more futureresponse team predictions.

BRIEF DESCRIPTION OF THE SEVERAL VIEW OF THE DRAWINGS

Having thus described some embodiments in general terms, references willnow be made to the accompanying drawings, which are not drawn to scale,and wherein:

FIG. 1 is a block diagram of an example responder prediction serversystem architecture within which at least some embodiments of thepresent invention may operate;

FIG. 2 is a block diagram of an example responder prediction servercomputing device structured in accordance with at least some embodimentsof the present invention;

FIG. 3 is a block diagram of an example client computing devicestructured in accordance with at least some embodiments of the presentinvention;

FIG. 4 is a flowchart diagram of an example process for training aresponder prediction machine-learning model in accordance with at leastsome embodiments of the present invention;

FIG. 5 is a flowchart diagram of an example process for determining aresponse team prediction object and rendering a response team suggestioninterface based on an alert in accordance with at least some of theembodiments of the present invention;

FIG. 6 is a flowchart diagram of an example process for training aresponder prediction machine learning model using a second responderprediction training corpus in accordance with at least some of theembodiments of the present invention;

FIG. 7 is a flowchart diagram of an example process for training aprioritization machine learning model using a responder predictiontraining corpus in accordance with at least some of the embodiments ofthe present invention;

FIG. 8 is a flowchart diagram of an example process for training aprioritization machine learning model using a second responderprediction training corpus in accordance with at least some of theembodiments of the present invention;

FIG. 9 is a flowchart diagram of an example process for determining aprioritization weight for an alert in accordance with at least some ofthe embodiments of the present invention;

FIG. 10 provides an operational example of a response team suggestioninterface in accordance with at least some of the embodiments of thepresent invention;

FIG. 11 provides exemplary training operations performed in accordancewith at least some of the embodiments of the present invention; and

FIG. 12 provides exemplary processing operations performed in accordancewith at least some of the embodiments of the present invention.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Various embodiments of the present invention will now be described morefully hereinafter with reference to the accompanying drawings, in whichsome, but not all embodiments of the disclosure are shown. Indeed, thedisclosure may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein. Rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. The term “or” is used herein in both the alternativeand conjunctive sense, unless otherwise indicated. The terms“illustrative,” “example,” and “exemplary” are used to be examples withno indication of quality level. Like numbers refer to like elementsthroughout.

Overview

Various embodiments of the present invention address technical problemsassociated with generating and transmitting response team predictionsfor routing incident alerts generated by alert monitoring tools ofcomplex platforms (e.g., monolithic software platforms and/orservice-oriented platforms). Modern complex platforms are supported byvast software development and IT teams. Such teams are consistentlychanging and developing new and varied expertise. Given that largecomplex platforms can produce over 1,000,000 incident alerts or cautionsper day, it is important that such incident alerts are quickly assessedand accurately routed to an appropriate response team.

Various embodiments discussed herein can be utilized by a responderprediction service. The responder prediction service is disposed incommunication with an alert monitoring service tool that is configuredto identify incident alerts generated by a complex platform. Theresponder prediction service may extract the alerts generated by thealert monitoring service tool and apply a responder prediction machinelearning model to process the alerts to generate a response teamprediction. The response team prediction may comprise an alertidentifier which may be used by a responder prediction enrichmentservice to generate a client identifier set associated with anappropriate software development or IT team (e.g., the appropriateresponder or response team data) associated with the monolith softwareplatform that is adept at handling the identified alert. The responderprediction service then proceeds to transmit the identified alert to theresponder prediction enrichment service, which may in turn transmit theenriched response team prediction to the client devices of theappropriate responder team based on the client identifier set.

The claimed invention is configured to produce response team predictionsefficiently and effectively and thereby reduce or eliminate manualrouting of alerts to responder teams. The claimed invention is furtherconfigured to reduce or eliminate errors in determining appropriateresponse team(s) and/or in routing alerts to such response teams.Various embodiments discussed herein further reduce the time needed todetermine appropriate response teams and thereby reduce the overall timeneeded for an issue to be detected and resolved. For instance, variousembodiments discussed herein may reduce the time needed to determineappropriate response teams to within 300 milliseconds from the incidentalert being received from an alert monitoring service tool.

In some embodiments, when detecting incident alerts on a complexplatform, various embodiments of the present invention enable: (i)returning relevant response team predictions based on incident alertsextracted from an alert monitoring service tool; (ii) efficient links toresponse team predictions based on incident alerts, such that properresponse team predictions are returned in less time than alternativeprocedures and means; (iii) integrating response team predictions into aprediction service API; (iv) rendering a response team suggestioninterface; and (v) testing responder prediction machine learning modelsiteratively with feedback, such that the responder prediction machinelearning model is updated iteratively throughout its life and throughoutthe growth cycle of the monolith software platform. In some embodiments,a service-oriented platform may be used by various embodiments of thepresent invention to enable: (i) returning relevant response teampredictions based on incident alerts extracted from an alert monitoringservice tool monitoring the service-oriented platform; (ii) efficientlinks to response team predictions based on incident alerts, such thatproper response team predictions are returned in less time thanalternative procedures and means; (iii) integrating response teampredictions into a prediction service API; (iv) rendering a responseteam suggestion interface; and (v) testing responder prediction machinelearning models iteratively with feedback, such that the responderprediction machine learning model is updated iteratively throughout itslife and throughout the growth cycle of the service-oriented platform.

Definitions

The term “complex platform,” or “application platform,” refer to asoftware platform comprising one or more types of software applications(e.g., monolithic platform and/or service-oriented platform), which maybe described in more detail below.

The terms “monolithic platform,” or “monolithic software platform,”refer to a software application designed to embody a single-tieredarchitecture in which the front end and back end systems are combinedinto a single platform. Monolithic software platforms are self-containedin that they can perform each operation needed to complete theirintended purpose or function. Such example monolithic platforms mayinclude Micros™ by Atlassian® platform or DynamoDB® by Amazon®.

The term “service-oriented platform” refers to a software applicationdesigned to embody a modular programming architecture based on specificservice types, wherein the modular programming may comprise existingservices combined by user specification in order to create a customsoftware application. In some embodiments, the services within themodular programming may configure GUI for user interaction with eachservice in an individual manner without affecting other services withinthe service-oriented platform.

A service-oriented platform is typically characterized by large networksof interdependent services and microservices that support a myriad ofsoftware features and applications. Indeed, some large service-orientedplatforms may be comprised of topologies of 1,500 or more interdependentservices and microservices. Such service-oriented platforms are nimble,highly configurable, and enable robust collaboration and communicationbetween users at individual levels, team levels, and enterprise levels.

Service-oriented platforms typically include large numbers of softwareapplications. Each software application includes a number of features,with many features (e.g., user authentication features) shared betweenmultiple software applications. Other features are supported only by oneassociated software application or a defined subset of softwareapplications.

A given service-oriented platform could support hundreds of softwareapplications and hundreds of thousands of features. Those applicationsand features could be supported by thousands of services andmicroservices that exist in vast and ever-changing interdependentlayers. Adding to this complexity is the fact that at any given time, agreat number of software development teams may be constantly, yetunexpectedly, releasing code updates that change various softwareservices, launch new software services, change existing features ofexisting software applications, add new software applications, add newfeatures to existing software applications, and/or the like. Stillfurther complexity is added by the fact that, at any given time, theconfigured rules with respect to directing alerts to a response team(e.g., by identifying a response team) may be incorrect or out-of-date,or the response team may comprise response team data that is out-of-date(e.g., team members or team ownership of the response team identified inthe response team prediction object has changed). Service-orientedplatforms may allow for such changes to be made within one or moreapplications and features within such applications.

The term “alert monitoring service tool” refers to a software servicethat is configured to monitor a complex platform (e.g., monolithicsoftware platform and/or service-oriented platform) and detect alerts,cautions, problems, errors, issues, or incidents. An example alertmonitoring service tool is Opsgenie® by Atlassian®. Alert monitoringservice tools may also generate one or more incident alerts (i.e.,alerts) of the complex platform (e.g., monolithic platform and/orservice-oriented platform) that may be extracted for application ofresponder prediction machine learning models. Such example alertmonitoring service tools may include SignalFX® by Splunk®, Opsgenie® byAtlassian®, or DynamoDB® by Amazon® in combination with AWS Lambda™ byAmazon®.

The term “alert” refers to a data object that is configured asinformation, text, and/or other media used to describe the operatingfunctionality of a complex platform (e.g., monolithic platform and/orthe service-oriented platform). Such operating functionality may includeindicators regarding the complex platform's performance (e.g., whetherthe complex platform and its functions are running at peak speed orslower than peak speed, if certain functions or capabilities are notrunning at peak performance or not running at all, etc.). Alert(s)include alert attributes as defined below. Alert(s) may be generated byan alert monitoring service tool and stored in a storage subsystem ofthe responder prediction server computing device. Alert(s) and/or alertattributes may be configured as a group or corpus of input data objectsthat are supplied as an input to train at least the responder predictionmachine learning model. Alert(s) associated with a complex platform(e.g., monolithic platform and/or service-oriented platform) are alsoused by a responder prediction machine learning model to generate aresponse team prediction.

The term “alert related dataset” refers to a collection of alerts andalert related data that are received from one or more alert monitoringservices tools over a predetermined time period. Alert related datasetsfurther comprise response team routing data for each alert identified inthe alert related dataset. The alert related dataset may be stored inthe storage subsystem of the responder prediction server system, such asthe responder prediction model repository.

The term “alert attribute” refers to data, text, identifiers, metadata,or other alert related characteristics or features that are extractedfrom alert related datasets and used to create a responder predictiontraining corpus as defined below. Alert attributes are extracted fromalert related datasets by a responder prediction service as definedbelow. Example alert attributes include an alert identifier, a tagidentifier, a log identifier, a description identifier, and a responderteam identifier. Example alert attributes, in some embodiments, mayfurther include a service identifier and a prioritization weightidentifier.

The term “alert identifier” refers to one or more items of data by whichan alert may be identified within a responder prediction server system.For example, an alert identifier may comprise text string(s), numericalcharacter(s), alphabetical character(s), alphanumeric code(s), ASCIIcharacter(s), a pointer, an IP address, a MAC address, a memory address,other unique identifier, or a combination thereof.

The term “responder prediction machine learning model” refers to amachine learning model that is trained and otherwise configured toreceive alerts (e.g., incident alerts) of a complex platform andgenerate response team predictions. The configuration data for acorresponding responder prediction machine learning model is stored on astorage subsystem associated with a responder prediction service. Theresponder prediction machine learning model may be trained using one ormore training corpuses (e.g., a responder prediction training corpus).Once the responder prediction machine learning model has been trained,the responder prediction machine learning model may process one or morealerts generated by an alert monitoring service tool and output aresponse team prediction to a prediction service API. Such exampleresponder prediction machine learning models may comprise a Naïve Bayesmachine learning model which is trained using the Bayes theorem and theassumption that each input variable is independent from the other inputvariables; a machine learning model trained to cluster the alerts; wordembedding using a Long Short-Term Model (LS™) and Natural LanguageProcessor (NLP); word embedding using an LS™, NLP, and a ConvolutionalNeural Network (CNN); and/or a graph-based model. In some embodiments, aNaïve Bayes machine learning model may be trained to predict theprobability of a phrase associated with n-grams, wherein certain phrasesmay contain a higher probability of being associated with a particularresponder team. In some embodiments, a clustering of alerts is based onembedded natural language values for each alert, wherein the alerts andthe associated embedded natural language values are similar for specificresponder teams such that only one responder team is identified for eachof the plurality of alerts within a cluster. In some embodiments, aplurality of Natural Language Processors (NLPs) and an LS™ may betrained to tokenize natural language values of the alert attributes andembed the phrases (e.g., words associated with a responder team) invectors associated with other similar phrases. In some embodiments, aplurality of NLPs, an LS™, and a sequential CNN may be trained totokenize the alert attributes using one of a plurality of NLPs which maybe associated with specific alert attributes, embedding each naturallanguage processed phrase from the alert attributes within an individualword vector graph for each alert attribute, and using a sequential CNNto concatenate each of the word vector graphs associated with the alertattributes. In some embodiments, a graph-based model may be used on thealert attributes, wherein various responder teams and users within theresponder teams may be graphed as nodes and may be connected to alertsbased on alert attributes (e.g., alert type identifier, tag identifier,description identifier, prioritization weight, etc.). In someembodiments, the graph-based model may be trained using graph neuralnetworks on previous alert attributes associated with previous responderteams and users of the responder teams. In addition, a deep neuralnetwork (DNN) may be used in the present invention as the responderprediction machine learning model, wherein two or more responderprediction machine learning models may be used to output a response teamprediction.

The term “response team prediction object” refers to a data object thatdescribes the dataset generated by the responder prediction machinelearning model for a corresponding alert. Response team predictionscomprise a client identifier set associated with client devices forresponse team members having expertise that is appropriate to address aparticular incident alert and/or a confidence score associated with theresponse team and response team members generated by the responderprediction machine learning model. In some embodiments, the responseteam prediction may be stored on the storage subsystem of the responderprediction server computing device.

The term “prioritization machine learning model” refers to a machinelearning model that is configured to process the output of the responderprediction machine learning model such that the response teampredictions are rearranged into a new order or sequence based on theassociated prioritization weights assigned to each alert. Theprioritization machine learning model may be configured to process thealerts and the response team predictions generated by the responderprediction machine learning model, in order to generate an operationsequence of an order to send out the response prediction alerts. Theprioritization machine learning model may be trained using extracteddata sets associated with past operation sequences (e.g., such as thoseoperation sequences created by a human operator or those operationsequences defined in a separate data object defining a complexplatform's specific policy). The prioritization machine learning modelmay be stored in a storage subsystem of the responder prediction servercomputing device such as the responder prediction model repository.

The term “prioritization weight identifier” refers to a descriptor of adata object associated with an alert that indicates the priority levelfor the alert. For example, an alert monitoring service tool identifiermay comprise text string(s), numerical character(s), alphabeticalcharacter(s), alphanumeric code(s), ASCII character(s), a pointer, an IPaddress, a MAC address, a memory address, other unique identifier, or acombination thereof. In some circumstances, prior prioritization weightidentifiers associated with prior alerts are processed by a machinelearning model (e.g., the prioritization machine learning model) suchthat the machine learning model may be trained to identify a predictedor suggested prioritization weight identifier for a new alert. Pastprioritization weight identifiers are associated with one or moreprioritization weights generated from a previous predetermined timeperiod (e.g., a high priority value, such as “P1,” may be associatedwith high priority alerts and lower priority values—such as “P2,” “P3,”and the like—may be associated with lower priority alerts). Theprioritization weight identifier may be stored in the storage subsystemof the responder prediction server system, like the responder predictionmodel repository.

The term “responder prediction training corpus” refers to data objectsthat are configured to train the one or more machine learning models ofthe responder prediction server system. Such training corpuses (e.g., afirst responder prediction training corpus, a second responderprediction training corpus, a third responder prediction trainingcorpus, etc.) may comprise data objects generated by a complex platformover a predetermined time period, wherein the training corpus may embodyprevious data objects associated with previously generated alerts. Forexample, a responder prediction training corpus may comprise alertattributes extracted from alert related datasets generated within apredetermined time period (e.g., a year). Such training corpuses may bestored in the storage subsystem and the responder prediction service ofthe responder prediction server system such as the responder predictionmodel repository.

The term “predetermined time period” refers to a data object thatdescribes a defined subset of time within a complex platform, such thatthe predetermined time period may be used to process only those alertsof the complex platform for that predetermined time period and anyassociated response teams generated from those alerts. In someembodiments, the predetermined time period may comprise multiple sets ofpredetermined time periods (e.g., a first predetermined time period, asecond predetermined time period, a third predetermined time period,etc.). The predetermined time period may be stored in the storagesubsystem of the responder prediction server system, such as theresponder prediction model repository, or in the prediction service APIof the responder prediction server system, such that alert relateddatasets may be generated based on the predetermined time period and thealert related datasets may have been stored in the storage subsystem ofthe responder prediction server system or in the prediction service APIof the responder prediction server system.

The term “description identifier” refers to one or more items of data bywhich a description of the alert may be generated by the complexplatform and captured by the alert monitoring service tool whenextracting the alert. For example, a description identifier may comprisetext string(s), numerical character(s), alphabetical character(s),alphanumeric code(s), ASCII character(s), a pointer, an IP address, aMAC address, a memory address, other unique identifier, or a combinationthereof.

The term “log identifier” refers to one or more items of data by which ahistorical log of current alerts, past alerts, and responder teams maybe captured by the alert monitoring service tool within a responderprediction server system (e.g., a log identifier may comprise data bywhich a log of actions taken on an alert are stored, which may includedata of an alert being acknowledged by one or more responder teams, dataof an alert being resolved by one or more responder teams, and/or dataof one or more responder teams being added to an alert). For example, alog identifier may comprise text string(s), numerical character(s),alphabetical character(s), alphanumeric code(s), ASCII character(s), apointer, an IP address, a MAC address, a memory address, other uniqueidentifier, or a combination thereof.

The term “tag identifier” refers to one or more items of data b which analert is tagged by the complex platform within a responder predictionserver system. For example, a tag identifier may comprise textstring(s), numerical character(s), alphabetical character(s),alphanumeric code(s), ASCII character(s), a pointer, an IP address, aMAC address, a memory address, other unique identifier, or a combinationthereof.

The term “service identifier” refers to one or more items of data bywhich a service may be identified as associated with specific responderteams within a responder prediction server system. For example, aservice identifier may comprise text string(s), numerical character(s),alphabetical character(s), alphanumeric code(s), ASCII character(s), apointer, an IP address, a MAC address, a memory address, other uniqueidentifier, or a combination thereof. In some embodiments, the serviceidentifier may comprise data indicating the upstream and downstreamservices for each specific service associated with specific responderteams. In some embodiments, the service identifier may comprise servicetier level data to indicate the importance of the service (e.g., theimportance of the service to a company/user using the complex platform)for the user of the complex platform (e.g., monolithic platform and/orthe service-oriented platform). In some embodiments, the serviceidentifier may comprise data associated with other service identifierswhich may indicate the number of impacted services from an alert of aspecific service.

The term “response team identifier” refers to one or more items of databy which a response team may be identified within a responder predictionserver system. For example, a response team identifier may comprise textstring(s), numerical character(s), alphabetical character(s),alphanumeric code(s), ASCII character(s), a pointer, an IP address, aMAC address, a memory address, other unique identifier, or a combinationthereof. The response team identifier may be used as a means to classifyspecific response teams for the response team prediction object.

The term “responder prediction service” refers to an application,program, platform, and/or software module configured for applying one ormore responder prediction machine learning models to one or more alertsof a complex platform to generate a response team prediction object. Theresponder prediction service may be configured to access a responderprediction model repository to access updated or newly trained respondermachine learning models.

The term “response team suggestion interface” refers to a graphical userinterface configured to indicate a response team prediction based on thedata objects and machine learning model(s) (e.g., responder predictionmachine learning model and/or prioritization machine learning model)stored in the responder prediction model repository.

DETAILED DESCRIPTION Example System Architecture

Systems, computer program products, and methods of the present inventionmay be embodied by any of a variety of devices. For example, the systemsand methods of an example embodiment may be embodied by a networkedcomputing device (e.g., an enterprise platform), such as a server orother network entity, configured to communicate with one or moredevices, such as one or more client devices, one or more user devices,and one or more external services. Additionally, or alternatively, thecomputing device may include fixed computing devices, such as a personalcomputer or a computer workstation. Still further, example embodimentsmay be embodied by any of a variety of mobile devices, such as aportable digital assistant (PDA), mobile telephone, smartphone, laptopcomputer, tablet computer, wearable computer, or any combination of theaforementioned devices.

FIG. 1 depicts an exemplary architecture 100 for generating a responseteam prediction object associated with an alert. The architecture 100includes one or more client computing devices 102 and a responderprediction server system 101. The responder prediction server system 101is configured to store an alert extractor unit, a responder predictionmodel repository 108, and responder prediction service 106.

In some embodiments, the responder prediction server system 101 isconfigured to train one or more machine learning models to generate aresponse team prediction object based on one or more extracted alertsfrom the alert extractor unit 113 which may be transmitted from an alertmonitoring service tool 151. The alert monitoring service tool 151 maybe configured to collect one or more alert related datasets over apredetermined time period. An alert extractor unit 113 may be configuredto extract the collected alert related dataset(s) and transmit said oneor more alert related datasets to the responder prediction service 106for processing and transmit the one or more alert related datasets to aresponder prediction machine learning model 155 for training. Theresponder prediction service 106 may be configured to generate aresponse team prediction object—which may be aggregated in a responseteam prediction data unit 114 to be transmitted to a response predictionAPI 171—based on a responder prediction machine learning model 155,which is trained using a response prediction model training unit 115.The response prediction model training unit 115 may be configured totrain one or more machine learning models (e.g., a responder predictionmachine learning model 155 and/or a prioritization machine learningmodel 175) using an alert related datasets unit 111 received from analert extractor unit 113. The prediction service API 171 may beconfigured to transmit the response team prediction data unit 114 to thea responder prediction enrichment service 181, which may enrich theresponse team object of the response team prediction data unit 114 withresponse team data (e.g., data associated with the response teamsincluding email addresses, names, and other correspondence information).In some embodiments, the responder prediction enrichment service 181 maybe housed within the client computing device 102 as an API, such thatthe prediction service API 171 may be in direct communication, over anetwork, with the client computing device 102 to transmit the responseteam prediction data unit 114. In some embodiments, the responderprediction enrichment service 181 of the client computing devices 102,may enrich the response team prediction object of the response teamprediction data unit 114 and configure the GUI of the client computingdevice 102, wherein the content of the configured GUI (e.g., responseteam suggestion interface) of the client computing device 102 maycomprise the response team prediction as a viewable, by a user of theclient computing devices 102, configuration of the response teamprediction object(s).

The responder prediction server system may be configured to provide aresponse team prediction object based on one or more extracted alerts(i.e., real-time alert data 152) from the alert extractor unit 113 whichmay be transmitted from an alert monitoring service tool 151. Theextracted alert may be processed by a responder prediction machinelearning model 155 trained by the response prediction model trainingunit 115. The responder prediction machine learning model 155 may beconfigured to output a response team prediction object which may beaggregated with other response team prediction objects associated withone or more alerts and stored in the response team prediction data unit114. The response team prediction data unit 114 comprising the one ormore response team prediction objects may be transmitted for storage tothe responder prediction model repository 108 along with the associatedalerts (e.g., alert data 152). The responder prediction model repository108 may be configured to store at least the responder prediction machinelearning model 155, the training data (e.g., the one or more alertrelated datasets) used by the responder prediction machine learningmodel 155, the alert data 152 (e.g., real-time alert data), and theresponse team prediction data unit 114 (e.g., the response teamprediction object) generated by the responder prediction machinelearning model processing the alert data.

The present invention may be further described by reference to FIGS. 1,2, 3, 4 , and 5. With respect to FIG. 1 , a client computing device 102may be provided to allow access by an end user to the responderprediction server system 101. The responder prediction server system 101may comprise system modules such as an alert extractor unit 113, analert monitoring service tool 151, a response prediction model trainingunit 115, a responder prediction service 106, a storage subsystem 108,and a prediction service API 171.

In some embodiments, an end user may access the responder predictionserver system 101 using the client computing device 102. In someembodiments, an alert monitoring service tool 151 may receive an alert(e.g., alert data 152) when a complex platform is not running at peakperformance, at regular predetermined intervals, or when one or morepredetermined criteria or complex platform attributes reachpredetermined levels. Such alerts may start the process hereindescribed, of processing the alert through a responder predictionservice 106 (e.g., a responder prediction machine learning model 155and/or prioritization machine learning model 175) and returning aresponse team prediction object to a prediction service API 171.

With respect to training the response predication service 106 and itsresponder prediction machine learning model 155, a response predictionmodel training unit 115 may comprise an alert related datasets unit 111.The alert related datasets unit 111 may be used to create the firsttraining corpus and/or a second training corpus which may both comprisea collected set of alert related datasets (e.g., alert related datasetsextracted from specific predetermined time periods or alert relateddatasets extracted from specific response teams). The responderprediction machine learning model 155 may be trained iteratively basedon the first training corpus and the second training corpus, andadditional training corpuses generated as the machine learning modelsprocess one or more alerts. Each training corpus comprises alerts andassociated response team prediction objects or manually determinedresponse team routes for those alerts.

Once the responder prediction machine learning model 155 has beentrained, an alert monitoring service tool 151 may extract alert data 152from a complex platform (which may be associated with the responderprediction machine learning model's training). The alert data 152 may beprocessed by the responder prediction service 106 such that it is inputinto the responder prediction machine learning model 155. Once theresponder prediction machine learning model 155 has processed the alertdata 152, the responder prediction machine learning model 155 may outputthe correct response team prediction object. The response teamprediction object of the response team prediction data unit 114 may bestored in the responder prediction model repository 108 along with thealert data extracted from the alert monitoring service tool 151.

In some embodiments, the response team prediction object(s) of theresponse team prediction data unit 114 may be sent back to the responseprediction model training unit 115 to further train the responderprediction machine learning model 155 by updating the cache of alertrelated datasets unit 111. In some embodiments, the response teamprediction object(s) of the response team prediction data unit 114 maybe processed by the response prediction model training unit 115 and thenpushed to a prediction service API 171, which may transmit the responseteam prediction object back to an end user or other user interface viathe client computer device 102 after enriching the response teamprediction object using the responder prediction enrichment service 181.

The client computing devices 102 and the responder prediction serversystem 101 may communicate over one or more networks. A network mayinclude any wired or wireless communication network including, forexample, a wired or wireless local area network (LAN), personal areanetwork (PAN), metropolitan area network (MAN), wide area network (WAN),or the like, as well as any hardware, software and/or firmware requiredto implement it (such as, e.g., network routers, etc.). For example, anetwork may include a cellular telephone, an 802.11, 802.16, 802.20,and/or WiMax network. Further, a network may include a public network,such as the Internet, a private network, such as an intranet, orcombinations thereof, and may utilize a variety of networking protocolsnow available or later developed including, but not limited toTransmission Control Protocol/Internet Protocol (TCP/IP) basednetworking protocols. For instance, the networking protocol may becustomized to suit the needs of the page management system. In someembodiments, the protocol is a custom protocol of JavaScript ObjectNotation (JSON) objects sent via a Web Socket channel. In someembodiments, the protocol is JSON over RPC, JSON over REST/HTTP, and thelike.

Exemplary Document Collaboration Server Computing Device

The responder prediction server system 101 may be embodied by one ormore computing systems, such as apparatus 200 shown in FIG. 2 . Theapparatus 200 may include processor 202, memory 204, input/outputcircuitry 206, and communications circuitry 208. The apparatus 200 maybe configured to execute the operations described herein. Although thesecomponents 202-208 are described with respect to functional limitations,it should be understood that the particular implementations necessarilyinclude the use of particular hardware. It should also be understoodthat certain of these components 202-208 may include similar or commonhardware. For example, two sets of circuitries may both leverage use ofthe same processor, network interface, storage medium, or the like toperform their associated functions, such that duplicate hardware is notrequired for each set of circuitries.

In some embodiments, the processor 202 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 204 via a bus forpassing information among components of the apparatus. The memory 204 isnon-transitory and may include, for example, one or more volatile and/ornon-volatile memories. In other words, for example, the memory 204 maybe an electronic storage device (e.g., a computer-readable storagemedium). The memory 204 may be configured to store information, data,content, applications, instructions, or the like for enabling theapparatus to carry out various functions in accordance with exampleembodiments of the present invention.

The processor 202 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. In some preferred and non-limiting embodiments,the processor 202 may include one or more processors configured intandem via a bus to enable independent execution of instructions,pipelining, and/or multithreading. The use of the term “processingcircuitry” may be understood to include a single core processor, amulti-core processor, multiple processors internal to the apparatus,and/or remote or “cloud” processors.

In some preferred and non-limiting embodiments, the processor 202 may beconfigured to execute instructions stored in the memory 204 or otherwiseaccessible to the processor 202. In some preferred and non-limitingembodiments, the processor 202 may be configured to execute hard-codedfunctionalities. As such, whether configured by hardware or softwaremethods, or by a combination thereof, the processor 202 may represent anentity (e.g., physically embodied in circuitry) capable of performingoperations according to an embodiment of the present invention whileconfigured accordingly. Alternatively, as another example, when theprocessor 202 is embodied as an executor of software instructions, theinstructions may specifically configure the processor 202 to perform thealgorithms and/or operations described herein when the instructions areexecuted.

In some embodiments, the apparatus 200 may include input/outputcircuitry 206 that may, in turn, be in communication with processor 202to provide output to the user and, in some embodiments, to receive anindication of a user input. The input/output circuitry 206 may comprisea user interface and may include a display, and may comprise a web userinterface, a mobile application, a query-initiating computing device, akiosk, or the like. In some embodiments, the input/output circuitry 206may also include a keyboard, a mouse, a joystick, a touch screen, touchareas, soft keys, a microphone, a speaker, or other input/outputmechanisms. The processor and/or user interface circuitry comprising theprocessor may be configured to control one or more functions of one ormore user interface elements through computer program instructions(e.g., software and/or firmware) stored on a memory accessible to theprocessor (e.g., memory 204, and/or the like).

The communications circuitry 208 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 200. In this regard, the communications circuitry 208may include, for example, a network interface for enablingcommunications with a wired or wireless communication network. Forexample, the communications circuitry 208 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally, or alternatively,the communications circuitry 208 may include the circuitry forinteracting with the antenna/antennae to cause transmission of signalsvia the antenna/antennae or to handle receipt of signals received viathe antenna/antennae.

It is also noted that all or some of the information discussed hereincan be based on data that is received, generated and/or maintained byone or more components of apparatus 200. In some embodiments, one ormore external systems (such as a remote cloud computing and/or datastorage system) may also be leveraged to provide at least some of thefunctionality discussed herein.

Exemplary Client Computing Device

Referring now to FIG. 3 , a client computing device may be embodied byone or more computing systems, such as apparatus 300 shown in FIG. 3 .The apparatus 300 may include processor 302, memory 304, input/outputcircuitry 306, and a communications circuitry 308. Although thesecomponents 302-308 are described with respect to functional limitations,it should be understood that the particular implementations necessarilyinclude the use of particular hardware. It should also be understoodthat certain of these components 302-308 may include similar or commonhardware. For example, two sets of circuitries may both leverage use ofthe same processor, network interface, storage medium, or the like toperform their associated functions, such that duplicate hardware is notrequired for each set of circuitries.

In some embodiments, the processor 302 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 304 via a bus forpassing information among components of the apparatus. The memory 304 isnon-transitory and may include, for example, one or more volatile and/ornon-volatile memories. In other words, for example, the memory 304 maybe an electronic storage device (e.g., a computer-readable storagemedium). The memory 304 may include one or more databases. Furthermore,the memory 304 may be configured to store information, data, content,applications, instructions, or the like for enabling the apparatus 300to carry out various functions in accordance with example embodiments ofthe present invention.

The processor 302 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. In some preferred and non-limiting embodiments,the processor 302 may include one or more processors configured intandem via a bus to enable independent execution of instructions,pipelining, and/or multithreading. The use of the term “processingcircuitry” may be understood to include a single core processor, amulti-core processor, multiple processors internal to the apparatus,and/or remote or “cloud” processors.

In some preferred and non-limiting embodiments, the processor 302 may beconfigured to execute instructions stored in the memory 304 or otherwiseaccessible to the processor 302. In some preferred and non-limitingembodiments, the processor 302 may be configured to execute hard-codedfunctionalities. As such, whether configured by hardware or softwaremethods, or by a combination thereof, the processor 302 may represent anentity (e.g., physically embodied in circuitry) capable of performingoperations according to an embodiment of the present invention whileconfigured accordingly. Alternatively, as another example, when theprocessor 302 is embodied as an executor of software instructions (e.g.,computer program instructions), the instructions may specificallyconfigure the processor 302 to perform the algorithms and/or operationsdescribed herein when the instructions are executed.

In some embodiments, the apparatus 300 may include input/outputcircuitry 306 that may, in turn, be in communication with processor 302to provide output to the user and, in some embodiments, to receive anindication of a user input. The input/output circuitry 306 may comprisea user interface and may include a display, and may comprise a web userinterface, a mobile application, a query-initiating computing device, akiosk, or the like.

In embodiments in which the apparatus 300 is embodied by a limitedinteraction device, the input/output circuitry 306 includes a touchscreen and does not include, or at least does not operatively engage(i.e., when configured in a tablet mode), other input accessories suchas tactile keyboards, track pads, mice, etc. In other embodiments inwhich the apparatus is embodied by a non-limited interaction device, theinput/output circuitry 306 may include may include at least one of atactile keyboard (e.g., also referred to herein as keypad), a mouse, ajoystick, a touch screen, touch areas, soft keys, and other input/outputmechanisms. The processor and/or user interface circuitry comprising theprocessor may be configured to control one or more functions of one ormore user interface elements through computer program instructions(e.g., software and/or firmware) stored on a memory accessible to theprocessor (e.g., memory 304, and/or the like).

The communications circuitry 308 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 300. In this regard, the communications circuitry 308may include, for example, a network interface for enablingcommunications with a wired or wireless communication network. Forexample, the communications circuitry 308 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally, or alternatively,the communications circuitry 308 may include the circuitry forinteracting with the antenna/antennae to cause transmission of signalsvia the antenna/antennae or to handle receipt of signals received viathe antenna/antennae.

It is also noted that all or some of the information discussed hereincan be based on data that is received, generated and/or maintained byone or more components of apparatus 300. In some embodiments, one ormore external systems (such as a remote cloud computing and/or datastorage system) may also be leveraged to provide at least some of thefunctionality discussed herein.

Example Data Flows and Operations Example Model Training Operations

Provided below are techniques for training at least a responderprediction machine learning model. In some embodiments, similartechniques may be used to train a prioritization machine learning modelwhich may be used in conjunction with the responder prediction machinelearning model to generate a response team prediction object.

FIG. 4 is a flowchart diagram of an example process 400 for performingoperations that are configured to train a responder prediction machinelearning model. Via the various operations of process 400, the responderprediction server system 101 can train one or more responder predictionmachine learning models to generate one or more response team predictionobjects based on one or more alerts (real-time alerts) in an efficientmanner without human interaction which may, in turn, be transmitted to aprediction service API 171 to generate a response team suggestioninterface for transmission to a client computing device 102.

The process 400 begins at operation 401 when the alert extractor unit113 extracts one or more alert related datasets from an alert monitoringservice tool 151. The alert related datasets may be comprised within analert related dataset unit 111. The alert related datasets may beextracted from a predetermined time period, such as a predetermined timeperiod from a previous time period. For example, a predetermined timeperiod may comprise 24 hours (such as the previous 24 hours leading upto the time of extraction of the alert related datasets), days (such asthe previous week leading up to the time of extraction of the alertrelated datasets), months (such as a time period of the last monthleading up to the time of extraction, the time period of the last 2months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9months, 10 months, 11 months, 12 months, or any time period within thoselisted herein), years (such as a time period of the last year, last twoyears, three years leading up to the time of extraction of the alertrelated datasets, or any time within those listed herein). In someembodiments, the selected predetermined time period may be selected fromany time in which the complex platform has been running (e.g., hascomprised a data object, a program, source code, etc.). In someembodiments, the selected predetermined time period may be selected fromany time in which an alert has been generated (either manually by ahuman operator or programmatically, such as by an Opsgenie® programcreated by Atlassian®) for the complex platform.

The alert related datasets extracted over this predetermined time periodmay comprise any alerts generated for the complex platform, includingany alerts created by a human operator of the complex platform, a tenantof the complex platform (e.g., a consumer request to fix an issueidentified by the tenant), a programmatic alerting program (e.g.,Opsgenie® created by Atlassian®), or semi-programmatically (e.g., analert generated by a program such as Opsgenie® by Atlassian®, which mayonly identify a source, wherein the alert is later identified by type bya human operator or tenant).

The alert related datasets extracted, or collected, by the alertextractor unit 113 may be transmitted to the responder predictionservice 106. At step 402, the responder prediction service 106 mayextract the alert attributes from the alert related datasets to create aresponder prediction training corpus. The alert attributes extractedfrom the alert related datasets may comprise an alert identifier, a tagidentifier, a description identifier, a log identifier, and a responderteam identifier. In some embodiments, the alert attributes may furthercomprise a prioritization weight identifier and/or a service identifier.

The alert identifier associated with the alert attributes of an alertrelated dataset may classify specific types of alerts of the alertrelated dataset, based on the unique identifiers of the alertidentifier. For example, the alert identifier may comprise CPU errors,storage errors (e.g., low storage availability), bug error preventingsource code from running, front-end API errors, security threats, loginerrors, tenant, integration errors, data transfer errors, or any othererrors known in the field. The alert identifier may be used within thealert related datasets to train the responder prediction machinelearning model to predict an alert identifier of an alert processed at afuture period of time, after training the responder prediction machinelearning model. The type of alert identifier classified within an alertrelated dataset may be used by a machine learning model, such as theresponder prediction machine learning model, to reduce the possibleresponder team predictions that the machine learning model may choosefrom. For instance, if an alert identifier for an alert related datasetcomprises a classification of a CPU error, the responder predictionmachine learning model may limit the possible response team predictionobjects to only those teams that may work on a CPU error or those teamsfamiliar with CPU errors. In some embodiments, each responder predictionmachine learning model may process every alert attribute and a consensusof the responder prediction objects generated by the responderprediction machine learning model may be used to predict a responderprediction team.

In some embodiments, a plurality of responder prediction machinelearning models may be clustered together to process sequentially orconcurrently with each other responder prediction machine learning modelto process specific alert attributes (e.g., a specific responderprediction machine learning model to process only one specific alertattribute). In some embodiments, the plurality of responder predictionmachine learning models may comprise one or more aggregation layersafter the training of the responder prediction machine learning modelsby one or more alert related datasets. In some embodiments, theaggregation layer may comprise one or more assigned weights and outputsfrom the responder prediction machine learning models, wherein theaggregation layer may take the input of one or more assigned weights andoutputs from training of the responder prediction machine learningmodels and convert the one or more weights and outputs into a singlefeature space. In some embodiments, the aggregation layer may comprise aprocess to aggregate the weights and outputs of the responder predictionmachine learning models by using a matrix dot product layer. In someembodiments, the aggregation layer may comprise a process to aggregatethe weights and outputs of the responder prediction machine learningmodel by using a fully connected neural network layer that performsnon-linear feature combination to place the weights and outputs into acommon feature space. In some embodiments, the aggregation layer maycomprise an output of one or more features (e.g., a common feature spacemay comprise the same number of cells—for example, the responderprediction machine learning models previously trained—as the input ofthe fully connected neural network) such that the dimension of theoutput feature is equal to the size of the aggregation layer. In someembodiments, the output feature may be used in the responder predictionmachine learning models to predict the response team predictionobject(s).

The service identifier associated with the alert attributes of an alertrelated dataset may classify the responder teams associated with thespecific service, the upstream and downstream services for the specificservice, the service tier level, and/or the number of other impactedservices associated with the specific service. The service identifiermay classify the responder team based on the type of error identified(e.g., alert identifier), and may further identify the service team(i.e., response team) associated with the service or type of error basedon the alert. For example, if the alert identifier is classified as aCPU error, the service identifier may likewise classify only CPUresponder teams. Based on the service identifier processed by theresponder prediction machine learning model, the responder predictionmachine learning model may limit the possible response team predictionobjects to only those teams that may work on a CPU error or those teamsfamiliar with CPU errors to select from for generating a responder teamprediction object. Likewise, if the alert identifier is classified asbug error preventing source code from running, front-end API errors,security threats, login errors, tenant, integration errors, datatransfer errors; the service identifier may also be classified as sourcecode teams or those familiar with the source code, front-end API teamsor those familiar with the front-end API, security teams or thosefamiliar with the security of the complex platform, login and tenantsupervising teams, or data integration teams or those familiar with dataintegration, respectively.

The response team identifier associated with the alert attributes of analert related dataset may identify, or classify, a specific team orspecific teams for each alert of the alert related datasets. Theresponse team identifier may comprise data objects identifying specificteams, team members, correspondence data of each team member, or otherspecific identifying information of a responder team that previouslyattended to one or more alerts. The response team identifier may beprocessed by a responder prediction machine learning model for trainingso that the responder prediction machine learning model may generate thesame or similar response team identifiers when processing an alert(e.g., a real-time alert). The alert attributes of the associated alertrelated datasets may be aggregated to generate a responder predictiontraining corpus.

At step 403, the responder prediction machine learning model may betrained using the training corpus of the associated alert relateddatasets. The responder prediction machine learning model may processeach of the alert attributes (alert identifier, log identifier,description identifier, tag identifier, and responder team identifier)in order to identify relationships and patterns between each alertattribute. By identifying the patterns between each alert attribute, theresponder prediction machine learning model may process an alertcomprising one, two, or three of the alert attributes to determine alikely responder team identifier of an alert (i.e., real-time alert).

At step 404, the responder prediction machine learning model, aftertraining with the training corpus, may be stored in a responderprediction model repository.

The responder prediction model repository 108 may be embodied as a datastorage device such as a Network Attached Storage (NAS) device ordevices, or as a separate database server or servers. The responderprediction model repository 108 may include information accessed orstored by the responder prediction service 106 to facilitate theoperations of the responder prediction server system 101. The responderprediction model repository 108 may further store a plurality of machinelearning models (e.g., the responder prediction machine learning modeland/or the prioritization machine learning model), alert relateddatasets used to train the machine learning models (e.g., one or moretraining corpuses), alert data, response team data (e.g., identifyingdata associated with specific response teams or members of responseteams) associated with the specific alert, or a response team predictiondata unit comprising a database of possible response teams andidentifying data associated with the possible response teams.

In some embodiments, the responder prediction machine learning model maybe further trained using a second training corpus comprising a secondalert related dataset, like the process depicted in FIG. 6 , comprisingprocess 600. The process of 600 begins at operation 601 with theresponder prediction service 106 collecting a second alert relateddataset from the alert extractor unit 113. At step 602, the responderprediction service 106 may extract a second set of alert attributes fromthe second alert related dataset in order to generate a second responderprediction training corpus. The second alert related dataset may beextracted based on an alert monitoring service tool (e.g., alertmonitoring service tool 151) over a second predetermined time period.For instance, the alert monitoring service tool may only transmit alertrelated datasets associated with a specific second predetermined timeperiod such as a predetermined time period taking place after thepredetermined time period of FIG. 4 . In some embodiments, the secondpredetermined time period may comprise a portion of the predeterminedtime period of FIG. 4 such that there is an overlap of time and data inthe alert related dataset and the second alert related dataset. In someembodiments, the second predetermined time period may comprise a timeperiod before the predetermined time period of FIG. 4 , such that thesecond alert related dataset comprises older in time alert attributeswhich may not be as up-to-date as the predetermined time period used totrain the responder prediction machine learning model of method 400.Similar to the predetermined time period of FIG. 4 , the secondpredetermined time period may also comprise time such as days, months,years, or any time within those periods. In some embodiments, the secondpredetermined time period may comprise a time period after thepredetermined time period of FIG. 4 , such that the second alert relateddataset comprises current alert attributes. In some embodiments, a usermay select, via a GUI, the second predetermined time period from aplurality of possible predetermined time periods.

The second alert attributes of the of the second alert related datasetsmay be extracted by the responder prediction service 106 to generate thesecond responder prediction training corpus. Similar to the alertattributes of FIG. 4 , the second alert attributes may also comprise analert identifier, a description identifier, a log identifier, a tagidentifier, and a responder team identifier for each associated alertrelated dataset of the second alert related datasets.

At step 603, the responder prediction machine learning model may betrained using the second responder prediction training corpus. At step604, the responder prediction machine learning model may be stored inthe responder prediction model repository. The responder predictionmodel repository may store the versions of the responder predictionmachine learning model as separate versions and track updates to theresponder prediction machine learning model as it is trained and storedto the responder prediction model repository.

An example training operation may be shown in FIG. 11 , wherein FIG. 11provides example training operations performed on the responderprediction machine learning model. In some embodiments, alert relateddatasets may be extracted from an alert monitoring service tool 151 at1101. The alert attributes of the alert related datasets may beextracted by a responder prediction service 106 at 1102. A responderprediction training corpus is generated at 1103 based on the extractedalert attributes. The responder prediction training corpus of 1103 isprocessed by the responder prediction machine learning model to trainthe responder prediction machine learning model at 1105. Once theresponder prediction machine learning model has been trained with theresponder prediction training corpus of 1103, the responder predictionmachine learning model is stored in the responder prediction modelrepository 108 at 1106. In some embodiments, an acknowledgement ofreceipt by the responder prediction model repository 108 may betransmitted to the responder prediction service 106 at 1107.

In some embodiments, the responder prediction service 106 may return astatus or acknowledgement of the responder prediction machine learningmodel to the monitoring service tool 151 at 1108A. In some embodiments,the responder prediction service 106 may request second alert relateddatasets at 1108B for further training of the responder predictionmachine learning model. In some embodiments, the monitoring service tool151 may automatically send the second alert related datasets to theresponder prediction service 106 in combination with the alert relateddataset of 1101. In some embodiments, the monitoring service tool 151may automatically send the second alert related datasets to theresponder prediction service 106 after the monitoring service tool 151has received the status of the responder prediction machine learningmodel at 1108A.

In some embodiments, the monitoring service tool 151 may extract secondalert related datasets and transmit the second alert related datasets tothe responder prediction service 106 at 1111. In some embodiments, theresponder prediction service 106 may extract second alert attributesfrom the second alert related datasets at 1112. Based on the secondalert attributes, the responder prediction service 106 may generate asecond responder prediction training corpus at 1113. The secondresponder prediction training corpus generated at 1113 by the responderprediction service 106 may be processed by the responder predictionmachine learning model at 1115 in order to train the responderprediction machine learning model. In some embodiments, once theresponder prediction machine learning model has been further trained bythe second responder prediction machine learning model at 1115, theresponder prediction machine learning model may be stored by theresponder prediction model repository 108 at 1116. In some embodiments,once the responder prediction machine learning model has been stored at1116, the responder prediction model repository 108 may return a statusor acknowledgement of the responder prediction machine learning model tothe responder prediction service 106 at 1117. In some embodiments, theresponder prediction service 106 may, after receiving acknowledgementfrom the responder prediction model repository 108, return the status ofthe responder prediction machine learning model at 1118A. In someembodiments, the status of the responder prediction machine learningmodel may comprise an acknowledgement from the responder predictionmodel repository 108 that the responder prediction service 106 isperforming and/or to indicate that an alert was processed successfully.In some embodiments, the status of the responder prediction machinelearning model may indicate that the responder prediction service 106 isnot performing and that an error was returned by one or more components(e.g., responder prediction machine learning model, prioritizationmachine learning model, response prediction model training unit, etc.)of the responder prediction service 106.

In some embodiments, a prioritization machine learning model 175 may betrained within the responder prediction server system 101. For example,and as shown in FIG. 7 , a prioritization machine learning model 175 maybe trained using responder prediction corpus of FIG. 4 , wherein theresponder prediction corpus comprising alert attributes furthercomprises an alert attribute of a prioritization weight identifier. Theprioritization weight identifier of the responder prediction corpus usedto train the prioritization machine learning model may compriseprioritization weights generated from a previous predetermined timeperiod (e.g., the predetermined time period of the responder predictiontraining corpus of FIG. 4 , or a different predetermined time periodthan the responder prediction training corpus of FIG. 4 ). Theprioritization weight may be used to identify certain orders orsequences of processing the alerts through the responder predictionmachine learning model 155 or the prioritization weight may be used toidentify certain orders or sequences of transmitting the alerts to theprediction responder teams (i.e., order to transmitting the responderteam prediction objects to the prediction service API and/or the clientcomputing devices). For example, a higher prioritization weight (e.g.,P1) associated with an alert processed by a responder prediction machinelearning model may be processed before an alert comprising a lowerprocessing weight (e.g., P2, P3, P4, etc.). Alternatively, theprioritization weights may be listed alternatively with P1 being thelowest, and the last alert processed, and any number above P1 comprisinghigher priority (e.g., P2 is higher than P1 but lower than P3, P3 ishigher than P2 but lower than P4, P4 is higher than P3 but lower thanP5, all the way up to PN).

In some embodiments, the prioritization machine learning model 175 maybe trained based on a prioritization weight identifier comprised withinan alert attribute of the responder prediction training corpus. Theprioritization weight identifier may review each of the alertsassociated with the alert related datasets of FIG. 4 , further includingthe associated prioritization weight identifiers, and based on at leastthe associated prioritization weight identifiers and the other alertattributes (i.e., alert identifier, service identifier, tag identifier,log identifier, description identifier, and responder team identifier),the prioritization machine learning model 175 may be trained to learnassociations between each of the alert attributes. For example, theprioritization machine learning model 175 may identify certainrelationships of the alert attributes, for each of the alerts within analert related dataset, to certain patterns between alert identifiers,service identifiers, tag identifiers, log identifiers, descriptionidentifiers, and/or responder team identifiers of the alert attributesand a prioritization weight identifier.

At step 702, the prioritization machine learning model 175 may be storedin the responder prediction model repository 108.

In some embodiments, the prioritization machine learning model 175 maybe trained with a second alert related dataset comprising second alertattributes. For example, and as shown in FIG. 8 , second alert relateddatasets may be extracted and collected from one or more monitoringservice tools (e.g., 151) and transmitted to a responder predictionservice 106, at step 801. The responder prediction service 106 mayextract second alert attributes from the second alert related datasetsin order to create a second responder prediction training corpus at step802. At step 803, the prioritization machine learning model 175 may betrained using the second responder prediction training corpus. Once theprioritization machine learning model has been trained with the secondresponder prediction training corpus, the prioritization machinelearning model 175 may be stored in the responder prediction modelrepository 108.

Example Model Processing Operations

Provided below are technique for generating a response team predictionobject from one or more responder prediction machine learning modelsand, optionally, one or more prioritization machine learning models.

FIG. 5 is a flowchart diagram of an example process 500 for performingoperations that are configured to determine a response team predictionobject and rendering a response team suggestion interface based on saidresponse team prediction object. Via the various operations of process500, the responder prediction server system 101 may train one or moremachine learning models, including a responder prediction machinelearning model and/or a prioritization machine learning model. Theresponder prediction server system 101 may use the trained machinelearning model(s) to process one or more alerts in order to generate aresponse team prediction object. Once a responder team prediction objecthas been generated, it may transmitted to a prediction service API 171,which may generate the response team prediction object and confidencescore based on the responder team prediction object and transmit theresponse team prediction object and confidence score to the responderprediction enrichment service 181 or directly to the client computingdevice 102 which may comprise the responder prediction enrichmentservice 181. In some embodiments, the responder prediction enrichmentservice 181 may configure a GUI of the client computing device withresponse team identifier data (e.g., data comprising response team name,response team participants, client computing device data of responseteam users, etc.) and may render a GUI with the response team identifierdata associated with the responder team prediction object. The GUIrendered by the responder prediction enrichment service 181 may compriseresponse team predictions as user-viewable configuration of the responseteam prediction object(s).

FIG. 5 depicts a flowchart diagram of an example process for determininga response team prediction object and rendering a response teamsuggestion interface based on an alert and the associated response teamprediction object. The responder prediction server system 101 isconfigured to store an alert extractor unit, a response prediction modeltraining unit 115, a responder prediction model repository 108, andresponder prediction service 106.

In some embodiments, the responder prediction server system 101 isconfigured to train the responder prediction machine learning model 155and prioritization machine learning model 175, which may both be used bythe responder prediction service 106, to process one or more alerts togenerate a response team prediction object, and stored in a responderprediction model repository 108. The alert monitoring service tool 151may be configured to collect one or more alerts (e.g., real-timealerts). The alert monitoring service tool 151 may then transmit thecollected one or more alerts to the responder prediction service 106, atstep 501. The responder prediction machine learning model 155, which maybe housed in the responder prediction service 106, may apply theresponder prediction machine learning model 155 to the one or morealerts at step 502. At step 503 and based on the processing of the oneor more alerts by the responder prediction machine learning model 155, aresponder team prediction object for each alert may be determined.

At step 504, a response team suggestion interface based on the responseteam prediction object is rendered by the responder prediction service106. In some embodiments, the response team suggestion interface may betransmitted to the client computing device 102 from a prediction serviceAPI in communication with the responder prediction service 106. In someembodiments, the prediction service API may do the rendering of theresponse team suggestion interface based on a transmitted response teamprediction object from the responder prediction service 106.

In some embodiments, a prioritization machine learning model 175 mayalso be applied to the one or more alerts to determine a prioritizationassociated with each alert. An example processing operation may be shownin FIG. 9 , wherein FIG. 9 provides receiving one or more responderprediction data objects from the responder prediction machine learningmodel 155 at step 901. The one or more outputs may be applied to aprioritization machine learning model to determine a prioritizationweight for each responder prediction data objects at step 902. Theprioritization machine learning model may process the one or moreresponder prediction data objects after the responder prediction machinelearning model, such that the output of the responder prediction machinelearning model 155 (e.g., responder prediction data object) is processedby the prioritization machine learning model 175. In such an embodiment,the prioritization machine learning model may generate a rearrangedsequence of the one or more response team prediction objects based onthe one or more alerts, wherein if an alert associated with the responseteam prediction object is assigned a higher priority (e.g., P1), thenthe response team prediction object may be rearranged to the top of thesequence of the one or more response team prediction objects andtransmitted to the prediction service API 171 and subsequently to theclient computing device 102 (or the responder prediction enrichmentservice 181), as a response team suggestion interface comprising aresponse team prediction, before other response team prediction objectsof lower priority are transmitted to the prediction service API 171.Alternatively, if the response team prediction object is assigned alower priority (e.g., P2, P3, P4, etc.), then the response teamprediction object may be rearranged to the bottom of the sequence of theone or more response team prediction objects and transmitted to theprediction service API 171 and subsequently to the client computingdevice (or the responder prediction enrichment service 181), as theresponse team suggestion interface comprising a response teamprediction, after other response team prediction objects of higherpriority have been transmitted to the prediction service API 171.

An example processing operation to generate a response team predictionobject and rendering a response team suggestion interface may be shownin FIG. 12 , wherein FIG. 12 provides example processing operationsperformed on the one or more alerts received from the alert monitoringservice tool 151. In some embodiments, one or more alerts are extractedby an alert monitoring service tool 151 and transmitted to the responderprediction service 106 at 1201. The one or more alerts are applied to aresponder prediction machine learning model at 1213 to generate one ormore response team prediction objects, at 1213 and 1214, respectively.In some embodiments, a prioritization machine learning model 175 may beapplied to the one or more alerts to generate a prioritization weight ofeach of the one or more alerts before the one or more alerts areprocessed by the responder prediction machine learning model 155. Insome embodiments, a prioritization machine learning model 175 may beapplied to the one or more response team prediction objects generatedfrom the responder prediction machine learning model 155 to generate aprioritization weight of each of the response team prediction objects.

In some embodiments, the responder prediction service 106 may store theone or more response team prediction objects in the responder predictionmodel repository 108, at 1215. The one or more response team predictionobjects stored in the responder prediction model repository 108 maycomprise a prioritization weight generated by a prioritization machinelearning model 175. In some embodiments, the prioritization weights ofthe associated response team prediction objects may determine a sequenceof storing the one or more response team prediction objects in theresponder prediction model repository 108 (e.g., a response teamprediction object with a higher prioritization weight may be storedbefore a response team prediction object with a lower prioritizationweight).

In some embodiments, the responder prediction model repository 108 maytransmit an acknowledgment of the update of the responder predictionmachine learning model stored in the responder prediction modelrepository 108 at 1216. In some embodiments, the responder predictionmodel repository 108 may transmit an acknowledgement of an update of theprioritization machine learning model stored in the responder predictionmodel repository 108 to the responder prediction service 106.

In some embodiments, the responder prediction service 106 may generate aresponse team suggestion interface at 1217 to for rendering on a clientcomputing device 102. In some embodiments, the responder predictionservice 106 may generate the response team suggestion interface at 1217and transmit the response team suggestion interface to the clientcomputing device 102 at 1218. In some embodiments, the responderprediction service 106 may generate the response team suggestioninterface at 1217 and transmit the response team suggestion interface toa prediction service API 171 that is in communication with a clientcomputing device 102. In some embodiments, the responder predictionservice 106 may generate the response team prediction object for each ofthe one or more alerts at 1214, and after transmitting the one or moreresponse team prediction objects to the responder prediction modelrepository 108 at 1215 and receiving acknowledgement at 1216, theresponder prediction service 106 may transmit the one or more responseteam prediction objects to the prediction service API 171 to generatethe response team suggestion interface. In some embodiments, theprediction service API 171 may transmit the generated response teamsuggestion interface to the client computing device 102. In someembodiments, feedback from the client computing device may betransmitted back to the responder prediction service 106 as a binaryindication that the response team prediction objects generated by theresponder prediction service 106 was correct or incorrect (e.g., a userof the client device may select a “button” configured on the GUI toindicate whether the response team prediction was correct—check—orincorrect—cross-mark (“X”)). A person of skill in the art willunderstand that any feedback function or configured GUI comprising afeedback interface may be used by the invention described herein toreceive user interaction data of the performance of the responderprediction service 106 and that the feedback function is not limited“buttons” comprising checks or cross-marks (e.g., “X”).

Example User Interface Configurations

FIG. 10 illustrates an example GUI configured on a client computingdevice 102 in accordance with a response team suggestion interface. Withrespect to FIG. 10 , a GUI of a client computing device 102 may beconfigured to indicate one or more response team predictions in theresponse team suggestion interface. The response team prediction(s) inthe response team suggestion interface may comprise indications of analert, or error, type at 1005 and 1015, wherein 1005 describes the typeof effort as “increased network accessibility” and 1015 describes thealert as a “login error” and further describes the region of the alertas the “EU region.”). In some embodiments, the response team suggestioninterface may further comprise the identification of the responseteam(s) for each of the one or more response team predictions (e.g.,1003, 1013, 1023, 1033). In some embodiments, the identification of theresponse team(s) for each of the one or more response team predictionsmay comprise one or more response teams that may be notified of thealert (e.g., 1013 comprises both the “DBA Team” and the “Platform Team”as response teams; 1023 and 1033 both identify a “DBA Team” and“Front-End Team”, respectively, for the associated response teamprediction). In some embodiments, the GUI may be configured on theclient computing device 102 to comprise buttons for user feedback foreach of the one or more response team predictions, wherein the buttonsmay comprise a positive indication of user feedback that the responseteam prediction is correct (e.g., a “check” as shown at 1061, 1062, and1063) or negative indication of user feedback that the response teamprediction is incorrect (e.g., an “X” as shown at 1051, 1052, and 1053).By indicating that a response team prediction is correct or incorrectvia GUI (e.g., through selection of the check or “X”) the clientcomputing device 102 may transmit the feedback to the responderprediction server system 101 for further training of the one or moremachine learning models within the responder prediction service 106(e.g., responder prediction machine learning model 155 and/orprioritization machine learning model 175).

In some embodiments, the response team prediction of the response teamsuggestion interface may comprise a temporal indication (e.g., 1006) ofwhen the alert was detected in the complex platform.

In some embodiments, the GUI of the client computing device 102 may beconfigured to indicate the prioritization weight (1002, 1012, 1022) foreach of the response team predictions within the response teamsuggestion interface. In some embodiments, the response team predictionsmay be organized based on the associated prioritization weights. Forexample, the response team predictions of FIG. 10 are organized with thehighest prioritization weight 1002 (“P1”) at the top of the GUI and thelowest prioritization weight 1022 (“P3”) at the bottom of the GUI.

It is to be understood the implementations are not limited to particularsystems or processes described which may, of course, vary. It is also tobe understood that the terminology used herein is for the purpose ofdescribing particular implementations only and is not intended to belimiting. As used in this specification, the singular forms “a”, “an”and “the” include plural referents unless the content clearly indicatesotherwise. Thus, for example, references to “an image” includes acombination of two or more images and references to “a graphic” includesdifferent types and/or combinations of graphics.

Although the present disclosure has been described in detail, it shouldbe understood that various changes, substitutions and alterations may bemade herein without departing from the spirit and scope of thedisclosure as defined by the appended claims. Moreover, the scope of thepresent application is not intended to be limited to the particularembodiments of the process, machine, manufacture, composition of matter,means, methods, and steps described in the specification. As one ofordinary skill in the art will readily appreciate form the disclosure,processes, machines, manufacture, compositions of matter, means,methods, or steps, presently existing or later to be developed thatperform substantially the same function or achieve substantially thesame result as the corresponding embodiments described herein may beutilized according to the present disclosure. Accordingly, the appendedclaims are intended to include within their scope such processes,machines, manufacture, compositions of matter, means, methods, or steps.

That which is claimed is:
 1. A computer-implemented method of training aresponder prediction machine learning model for generating response teampredictions comprising: collecting alert related datasets originatingfrom one or more alert monitoring service tools over a predeterminedtime period; extracting alert attributes from the alert related datasetsto create a responder prediction training corpus, wherein the alertattributes comprise an alert identifier, a tag identifier, a logidentifier, a description identifier, and a responder team identifier;training the responder prediction machine learning model using theresponder prediction training corpus; and storing the responderprediction machine learning model following training to a responderprediction model repository, wherein the responder prediction modelrepository is accessible by a responder prediction service.
 2. Thecomputer-implemented method of claim 1 further comprising: collectingsecond alert related datasets originating from the one or more alertmonitoring service tools over a second predetermined time period;extracting second alert attributes from the second alert relateddatasets to create a second responder prediction training corpus;training the responder prediction machine learning model using thesecond responder prediction training corpus; and storing the responderprediction machine learning model following training to the responderprediction model repository.
 3. The computer-implemented method of claim1, further comprising: receiving one or more alerts from an alertmonitoring service tool; and applying, for each of the one or morealerts, a responder prediction machine learning model to determine aresponse team prediction object for each alert.
 4. Thecomputer-implemented method of claim 3, further comprising applying ascore to each response team prediction object of the one or more alerts.5. The computer-implemented method of claim 4, further comprisingdetermining the score of the response team prediction object using atleast one of a user input or a closing alert, and wherein the score iscalculated by comparing the response team prediction object with atleast one of the user input or the closing alert.
 6. Thecomputer-implemented method of claim 4, further comprising training theresponder prediction machine learning model in a subsequent stage usingthe score associated with each response team prediction object of theone or more alerts.
 7. The computer-implemented method of claim 6,wherein the score is applied to the responder prediction machinelearning model to determine one or more future response team predictionobjects.
 8. The computer-implemented method of claim 1, furthercomprising training a prioritization machine learning model comprising:training the prioritization machine learning model using the responderprediction training corpus, the alert attributes of the responderprediction training corpus further comprising a prioritization weightidentifier; and storing the prioritization machine learning modelfollowing training to the responder prediction model repository, whereinthe responder prediction model repository is accessible by a responderprediction service.
 9. The computer-implemented method of claim 8,further comprising: collecting second alert related datasets originatingfrom the one or more alert monitoring service tools over a secondpredetermined time period; extracting second alert attributes from thesecond alert related datasets to create a second responder predictiontraining corpus; training the prioritization machine learning modelusing the second responder prediction training corpus; and storing theprioritization machine learning model following training to theresponder prediction model repository.
 10. An apparatus for generating aresponse team prediction associated with one or more alerts, theapparatus comprising at least one processor and at least one memoryincluding program code, the at least one memory and program codeconfigured to, with the processor, cause the apparatus to at least:receive one or more alerts from an alert monitoring service tool; foreach of the one or more alerts, apply a responder prediction machinelearning model to determine a response team prediction object for eachalert; and cause rendering of a response team suggestion interface basedon the response team prediction object.
 11. The apparatus of claim 10,wherein the response team prediction object is transmitted to aprediction service API that is configured to indicate an alertnotification comprising at least one of the response team prediction, adataset of routing information associated with at least a clientidentifier set for the response team prediction, or the alert associatedwith the response team prediction.
 12. The apparatus of claim 10,wherein the responder prediction machine learning model comprises apre-training with an extracted alert related dataset associated with acomplex platform.
 13. The apparatus of claim 12, wherein the extractedalert related dataset comprises data extracted from a predetermined timeperiod.
 14. The apparatus of claim 10, wherein the at least one memoryand program code configured to, with the processor, cause the apparatusto at least: receive one or more alerts from an alert monitoring servicetool; and for each of the one or more alerts, apply a prioritizationmachine learning model to determine a prioritization weight for eachalert.
 15. The apparatus of claim 14, wherein an operation sequence ofprocessing for the responder prediction machine learning model isapplied to the one or more alerts based on the prioritization weight foreach of the one or more alerts.
 16. The apparatus of claim 14, whereinan operation sequence for determining the response team predictionobject is applied to the alerts based on the prioritization weight foreach alert.
 17. The apparatus of claim 14, wherein an operation sequencefor the rendering of the response team suggestion interface based on theresponse team prediction object is based on the prioritization weightfor each of the one or more alerts used to generate the response teamprediction object.
 18. The apparatus of claim 10, wherein a score isdetermined by the response team prediction associated with an alert andat least one of user input or a closing alert.
 19. The apparatus ofclaim 18, wherein the score is applied to the responder predictionmachine learning model to determine one or more future response teampredictions.